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© A visual information system to capture an input 
image using a camera 22, manipulate the image 
using processor 28, and project the processed im- 
age using optics 33 to superimpose the processed 
image on the actual object being observed by a 
viewer. Processing is done in real-time to allow the 



viewer to see both the actual and processed images 
while the viewer moves and changes viewing angles. 
Areas of interest in the displayed image may be 
highlighted or include graphical information for the 
viewer. 
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1. FIELD OF THE INVENTION 

This invention relates to the field of image 
processing systems, more particularly to head- 
mounted image processing and display systems. 

2. BACKGROUND OF THE INVENTION 

There are many instances when it is advanta- 
geous to use photographs or electronically stored 
images to aid a viewer in observing the features 
and spatial relationships of an object. The features 
the viewer would like to locate or see are some- 
times obscured or are difficult to locate with the 
unaided eye. 

One example is during a surgical procedure. A 
surgeon may take a photograph or an electronic 
image of an area before surgery begins. The image 
may be from previous diagnostic test such as 
magnetic resonance imaging (MR!) or computer- 
ized axial tomography (CAT). The surgeon will refer 
to this image during surgery as a guide for the 
procedure. For example, an image may be taken of 
a tumor before starting a removal procedure. Dur- 
ing the procedure, the surgeon can refer back to 
the image to assist in locating the tumor or to 
ensure that the entire tumor has been removed. 

SUMMARY OF THE INVENTION 

The present invention provides a method and 
device for providing visual information to a viewer 
without blocking the vision of the viewer or requir- 
ing the viewer to look away from the object of 
interest. One embodiment of the invention to pro- 
vides an imaging system that displays a computer 
generated image in the field of view of the user, 
without obstructing the view of the user. This allows 
the user to see the image without looking away 
from the object of interest. Furthermore, the image 
the user sees will be manipulated to have the 
correct perspective or viewpoint. The computer 
generated image may be superimposed on the 
actual image that the user sees to help the user 
correlate information from the computer generated 
image with the users actual view of the object. 

It is a further advantage of the invention that 
the movement of the user or object can be mon- 
itored by the system and the displayed images 
changed to compensate for the movement. This 
allows the user to see the object from different 
perspectives by simply moving his head. This 
three dimensional effect is important in that the 
image may show internal features that may not be 
visible by the user. For example, during a surgical 
procedure locations of major arteries can be 
marked to allow the surgeon to avoid them. 



BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a schematic view of one embodiment 
of a visual information system. 
5 FIG. 2 is a schematic view of a first embodi- 

ment of a display mechanism of the visual informa- 
tion system. 

FIG. 3 is a pictorial view of a second embodi- 
ment of a display mechanism of the visual informa- 
nt) tion system. 

FIG. 4 is a pictorial view of one embodiment of 
the visual information system. 

FIG. 5 is a pictorial view of a patients hand as 
seen through one embodiment of the disclosed 
15 visual information system. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

20 There are several drawbacks associated with 
using stored images to assist a viewer in identify- 
ing objects in his view. First, the viewer must look 
away from the object of interest in order to see the 
stored images, interrupting the work being per- 

25 formed and breaking the viewers concentration. 
Second, the images presented to the viewer have a 
fixed perspective, that is the viewer cannot usually 
manipulate the viewpoint of the image. Depending 
on the circumstances, the image may be difficult to 

30 correlate with the prespective of the viewer. For 
example, when testing an electronic assembly, the 
image of a component to be probed or removed 
may not be of much help when the component is 
surrounded by a large number of very similar, or 

35 worse yet, identical components. 

A much better solution is to superimpose the 
image on the object being viewed, in effect high- 
lighting the features of interest in the object. There 
are many applications for the vision enhancement 

40 imaging system disclosed, for example, three di- 
mensional information from the images used to 
guide the surgical procedure discussed above 
could be used to generate enhanced images and 
projected into the line of sight of the surgeon to 

45 allow the surgeon to see the images without having 
to look away from the patient. 

Figure 1 shows a schematic view of one em- 
bodiment of the disclosed imaging system. The 
video camera 22 monitors the subject, in this case 

so a surgery patient, and transmits an electronic re- 
presentation of the image to the processor 28 via 
channel 34. The choice of video camera depends 
on the information required and is not critical to the 
invention. For example, the camera could be mon- 

55 ochrome or color, digital or analog, and could be 
receptive to light outside the visible spectrum. In 
some applications it may be advantageous to have 
more than one camera to provide the processor 
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with additional perspectives to aid in the generation 
of three-dimensional images. 

Optional patient position sensor 24 transmits 
information to the processor 28 to allow the image 
processing and display functions to compensate for 
patient motion. Note that the object position and 
motion information can be extracted from the im- 
age obtained via the camera and the use of vision 
algorithms. The processor 28 may determine posi- 
tion and motion information by tracking features of 
the patient. If no suitable features exist naturally on 
the patient, high contrast marks could be drawn on 
the patient prior to beginning a procedure to allow 
the patient to be tracked optically by the processor 
28. 

Viewer position sensor 26 tracks the position of 
the viewer to allow the viewer to move or turn to 
see different views of an image. Not only does the 
position sensor 26 allow the processor 28 to com- 
pensate for the viewing angle, it allows the proces- 
sor 28 to scale the image as the distance from the 
object to the viewer changes and to move the 
image across the screen as the viewer scans the 
object. If the initial positions of the viewer and 
object are known, then motion sensors may be 
used instead of position sensors. Motion compen- 
sation prevents images from being warped or 
smeared as the patient moves. 

The processor 28 applies image processing 
algorithms to manipulate the image from the cam- 
era 22 and other image sources. The other sources 
are not shown in Figure 1, but could include 
scanned photographs, MRI or CAT scans, other 
medical diagnostic equipment, prior surgical proce- 
dures, or any image device. Image processing 
functions that would be desirable in some applica- 
tions are image conditioning, feature extraction, 
and matching or correlation algorithms. The com- 
puter generated images may be wire-frame repre- 
sentations or fully rendered three-dimensional im- 
ages or a hybrid combination of the two. For exam- 
ple, a tumor may be fully rendered and colorized 
while the surrounding tissue is represented by a 
wire-frame model. The processor could also use 
false coloring to highlight certain features or ob- 
jects. Textual data may also be displayed to inform 
the surgeon of the patients vital signs, duration of 
surgery, etc. 

In addition to manipulating the image data, the 
processor 28 may also receive inputs from the 
surgeon to allow the surgeon to mark areas of 
interest and to build a graphical database for later 
analysis. Inputs from the surgeon could also be 
used with stored information to aid feature recogni- 
tion. The surgeon could use any available means to 
input data into the processor including a keyboard, 
mouse, touch screen, trackball, light pen, or voice 
recognition. 



Processor 28 can be any type of processor or 
group of processors capable of performing the 
necessary image processing and display control 
functions required of the imaging system. For ex- 
5 ample, the processor could be designed around 
the Texas Instruments Multimedia Video Processor 
(MVP) image processing chip. The processor 
should have sufficient through-put to process the 
images, object motion, viewer position, and viewer 

;o input data in real-time. The required frame rate will 
vary depending on the application but should pref- 
erably be in the range of 60 to 100 frames per 
second, although 30 or fewer frames per second 
may be acceptable in some circumstances, espe- 

75 cially situations with very little movement. The 
frame rate may be dynamically varied as image 
complexity and image processing algorithms vary 
the demands on processing power. 

The processed images are sent to the display 

20 device 30 via display control bus 40. The display 
device could be any type of spatial light modulator 
capable of displaying an image such that it appears 
superimposed on an object being viewed. Optics 
33 are used as required to focus and project the 

25 image. Light source 32 and optics 33 may not be 
required in all applications depending on the type 
of spatial light modulator used. 

One embodiment of a display system 68 com- 
prising the display device, light-source, and optics 

30 is shown in Figure 2. Light from source 54 is 
focused and collimated as necessary by lens 58. 
The focused light 56 is modulated by a spatial light 
modulator, in this case a DMD array 60. The re- 
flected light 62 is focused and magnified as neces- 

35 sary by lens 64. The light is then reflected off of 
the lens 50. At the same time, light 48 from the 
object 46 is allowed to pass through the lens 50. 
The light 52 from the object and the DMD array 
exits the display system 68 and is seen by the 

40 viewer. 

The lens 50 may be worn like a pair of gog- 
gles, or a face shield, to allow the viewer to see the 
object 46 through the lens 50 and also see the 
image projected onto the lens. Lens 50 may be 

45 partially silvered to reflect light or may simply be 
any material with a partially reflective surface. Lens 
50 is typically designed to allow approximately 
90% of the light from the object to pass through 
the lens 50 while reflecting 10% of the reflected 

so light 66 from the DMD. The amount of light re- 
flected and transmitted by lens 50 depends on the 
application and the amount of light available from 
the object and the imager. The DMD 60 is particu- 
larly useful for this applications because it allows 

55 the brightness of the projected image to be ad- 
justed over a wide range. This wide adjustment 
allows the viewer to strike an acceptable balance 
between image brightness and object visibility. 
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A second embodiment of a display system 
according to the present invention simply attenu- 
ates light from the object of interest. This second 
embodiment, shown in Figure 3, may use a liquid 
crystal display (LCD) instead of a DMD. Because 
no light source or projection optics are required, 
the LCD 67 may be the actual lens of the goggles 
69 worn by the viewer. Cable 71 carries the signals 
necessary to operate the LCD. The use of an LCD 
instead of the reflective lens of Figure 2 attenuates 
more of the light from the image and reduces the 
visibility of the viewer. Areas of interest may be 
marked by attenuating light from the areas of inter- 
est, or by attenuating light from all areas except 
those of interest. 

A pictorial view of one embodiment of the 
surgical imaging system is shown in Figure 4. 
Viewer 72 looks though display system 68 at the 
object 46. In this embodiment both the display 
system 68 and the viewer position sensor 26 are 
head mounted on the viewer 72. Object position 
sensor 24 is attached to the object 46 without 
obstructing the view of either the viewer 72 or the 
camera 22. Processor 28 receives data from the 
camera 22, the viewer position sensor 26, and the 
object position sensor 24 and sends data to the 
display system 68. 

Figure 5 shows one example of the view 
through the imaging system during an operation to 
remove a foreign object from a patient. In Figure 5 
a surgeon is attempting to remove a foreign object 
80 from a patient's hand 82. Textual information 84, 
such as the patient's vital signs, distances between 
objects, and elapsed time may be displayed. Ob- 
jects of concern, in this case a nerve 86 that the 
surgeon is attempting to avoid, and the entrance 
path 88 of the object 80, may be highlighted. It 
should be appreciated that as the surgeon moves 
in relation to the patient, the perspective of the 
displayed image and the patient's hand will change 
in unison allowing the surgeon to determine the 
best method of removing the object 80. 

Thus, although there has been disclosed to this 
point a particular embodiment for a surgical imag- 
ing system, it is not intended that such specific 
references be considered as limitations upon the 
scope of this invention except in-so-far as set forth 
in the following claims. Furthermore, having de- 
scribed the invention in connection with certain 
specific embodiments thereof, it is to be under- 
stood that further modifications may now suggest 
themselves to those skilled in the art, it is intended 
to cover all such modifications as fall within the 
scope of the appended claims. 



Claims 

1. An imaging system comprising: 

a camera to monitor a scene in order to 
5 convert scene information into electronic sig- 

nals, and to transmit said signals; 

a processor to receive said signals from 
said camera, and to perform image processing 
routines on said signals, and to generate pro- 
w cessed signals representing information about 

said scene; and 

a display device to receive said processed 
signals and to convert said processed signals 
into a displayed image, wherein a viewer may 
15 look through said display device and see both 

said scene and said displayed image. 



The imaging system of claim 1, wherein said 
displayed image is superimposed on said 
scene from the perspective of said viewer. 
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3. The imaging system of claim 1, wherein said 
displayed image is projected onto the lens of a 
pair of goggles. 

4. The imaging system of claim 1, wherein said 
displayed image is projected onto a face 
shield. 

5. The imaging system of claim 1 wherein said 
display device comprises a spatial light 
modulator, wherein said viewer looks through 
said spatial light modulator. 

6. The imaging system of claim 1 wherein said 
display device comprises a digital micromirror 
device. 

7. The imaging system of claim 1 wherein said 
display device comprises a liquid crystal dis- 
play. 

8. The imaging system of claim 1 wherein said 
display device comprises a partially silvered 
lens. 

9. The imaging system of claim 1, wherein said 
display devil is head-mounted on the user of 
said system. 

10. The imaging system of claim 1, wherein said 
processor performs said image processing 
routines in real-time. 

11. The imaging system of claim 1, wherein said 
processor tracks features of said scene such 
that motion in said scene is compensated. 



4 



EP 0 665 686 A2 



12. The imaging system of claim 1, further com- 
prising a position sensor attached to an object 
in said scene and in communication with said 
processor to allow said processor to monitor 
the position of said object in said scene. 

13. The imaging system of claim 1, further com- 
prising a motion sensor attached to an object 
in said scene and in communication with said 
processor to allow said processor to compen- 
sate for motion of said object within said 
scene. 

14. The imaging system of claim 1, further com- 
prising a position sensor attached to said user 
and in communication with said processor to 
allow said processor to compensate for move- 
ment of said user. 

15. The imaging system of claim 1, further com- 
prising a motion sensor attached to said user 
and in communication with said processor to 
allow said processor to compensate for move- 
ment of said user. 

16. The imaging system of claim 1, wherein said 
displayed image is a high-contrast picture of 
said scene. 



70 



tronic signals using a camera; 

tracking the position of said object; 

tracking the position of a viewer; 

processing said electronic signals to com- 
pensate for the relative position of said object 
and said viewer; 

converting said processed electronic sig- 
nals into an image; and 

displaying said image such that said view- 
er sees said image superimposed on said ob- 
ject. 



20. The method of claim 15 wherein tracking the 
position of said object using a position sensor 

75 is replaced with tracking the motion of said 

object with a motion sensor. 

21. The method of claim 15 wherein tracking the 
position of said viewer using a position sensor 

20 is replaced with tracking the motion of said 

viewer with a motion sensor. 
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17. The imaging system of claim 1, wherein said 
displayed image is a false-color representation 
of a portion of said scene. 
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18. An imaging system comprising: 

a camera to constantly monitor a scene 
and convert scene information to electronic 
signals; 

a processor to receive said electronic sig- 
nals and perform image processing routines on 
said electronic signals and to output a pro- 
cessed video signal; 

a position sensor attached to an object in 
said scene to communicate information con- 
cerning the position of said object to said 
processor; 

a position sensor attached to a user of 
said system to communicate information con- 
cerning the position of said user to said pro- 
cessor; 

a display device to receive said video sig- 
nal and to convert said video signal into a 
display image, wherein a viewer may look 
through said display device and see the dis- 
play image superimposed on said scene. 

19. A method of displaying an image of an object 
comprising: 

converting light from an object to elec- 
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FIG. 2 
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